Nov 2024 schema #32

fivetran-reneeli · 2025-01-31T17:33:02Z

PR Overview

This PR will address the following Issue/Feature: #28

This PR will result in the following new package version: v0.5.0-a1

Schema changes from Nov 2024

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

to be completed

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

dbt run –full-refresh && dbt test
dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

The appropriate issue has been linked, tagged, and properly assigned
All necessary documentation and version upgrades have been applied
docs were regenerated (unless this PR does not include any code or yml updates)
BuildKite integration tests are passing
Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

If you had to summarize this PR in an emoji, which would it be?

💃

models/apple_store__app_version_report.sql

…sistency test, update changelog

fivetran-joemarkiewicz

@fivetran-reneeli thanks for this PR. A few comments below following this review.

models/apple_store__app_version_report.sql

…d update models

fivetran-reneeli · 2025-02-05T20:45:28Z

models/intermediate/int_apple_store__date_spine.sql

+
+{% set first_date_query %}
+
+    select min(date_day) as min_date_day


If this makes sense, I will add the subscription staging models too. Just wanted to have a proof of concept first before spending time adding the subscription logic.

So the idea here is to get a date spine generated using the earliest date across all events and then use that earliest date as the starting point? I think this is a good approach!

However, to brainstorm a bit more - is there a simper way we could achieve this by using one table? This could be a performance update in the future, but I want to think through this. Can we make the claim that crashes and engagements wouldn't come until there is a download and therefore could just rely on the min date of the download daily? I'm not saying that's true, but want to think through any of these simpler possibilities.

Not required for pre-release, but wanting to bump this question to ensure it doesn't get lost.

Co-authored-by: Joe Markiewicz <[email protected]>

packages.yml

Co-authored-by: Joe Markiewicz <[email protected]>

fivetran-joemarkiewicz

A few small change requests before approval

fivetran-joemarkiewicz · 2025-02-19T15:08:15Z

CHANGELOG.md

@@ -1,4 +1,14 @@
-# dbt_apple_store version.version
+# dbt_apple_store v0.5.0-a1  


Same comment from the source to create a duplicate entry for the official v0.5.0 release while also retaining the v0.5.0-a1 release for posterity.

fivetran-joemarkiewicz · 2025-02-19T15:10:03Z

README.md

@@ -54,7 +55,7 @@ Include the following apple_store package version in your `packages.yml` file:
 ```yaml
 packages:
  - package: fivetran/apple_store
-    version: [">=0.4.0", "<0.5.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: 0.5.0-a1


This needs to now reflect what will be the updated official range.

fivetran-joemarkiewicz · 2025-02-19T15:10:20Z

README.md

@@ -146,7 +147,7 @@ This dbt package is dependent on the following dbt packages. These dependencies
 ```yml
 packages:
    - package: fivetran/apple_store_source
-      version: [">=0.4.0", "<0.5.0"]
+      version: v0.5.0-a1


This will need to now be the updated official range of the source

fivetran-joemarkiewicz · 2025-02-19T15:11:05Z

CHANGELOG.md

+# dbt_apple_store v0.5.0-a1  
+[PR #32](https://github.com/fivetran/dbt_apple_store/pull/32) includes the following updates:
+
+## Breaking Changes: Schema Change


I would also add in here the bugfix of materializing the reporting grains as tables. This will result in extra tables, but it's important to highlight why this change was made.

good point!

fivetran-joemarkiewicz · 2025-02-19T15:12:17Z

packages.yml

+  - git: https://github.com/fivetran/dbt_apple_store_source.git
+    revision: nov_2024_schema
+    warn-unpinned: false
+
+  # - package: fivetran/apple_store_source
+  #     version: 0.5.0-a1


Reminder that this will need to be updated before merge and release

fivetran-joemarkiewicz

LGTM

fivetran-joemarkiewicz · 2025-02-19T20:50:06Z

packages.yml

+  - git: https://github.com/fivetran/dbt_apple_store_source.git
+    revision: nov_2024_schema
+    warn-unpinned: false
+
+  # - package: fivetran/apple_store_source
+  #     version: 0.5.0


Reminder to swap before merge and release

CHANGELOG.md

fivetran-avinash · 2025-02-19T22:52:04Z

CHANGELOG.md

+- For more information on the upstream breaking changes concerning the source tables, refer to the [source package pre-release notes](https://github.com/fivetran/dbt_apple_store_source/releases/tag/0.5.0-a1).
+- The reporting grains are created in upstream intermediate models (found in the `intermediate/reporting_grain` folder). Along with the date spine (`int_apple_store__date_spine`), these reporting grain models are materialized as tables to enhance performance.
+
+## Documentation


Looks like we updated the DECISIONLOG, we should probably include a note here about these updates (or whereever appropriate).

thank you good point!

CHANGELOG.md

fivetran-avinash · 2025-02-19T22:57:45Z

CHANGELOG.md

+- Following the connector's [Nov 2024 Update](https://fivetran.com/docs/connectors/applications/apple-app-store/changelog#november2024) to sync from the [App Store Connect API](https://developer.apple.com/documentation/appstoreconnectapi), we've updated this dbt package to reflect the new schema which includes the following changes:
+
+# Breaking Changes
+- The `account_id` and `account_name` fields have been removed.


We should call out the models these fields have been removed from--based on the yml updates, it looks like it's just the subscription report?

that's correct, updated!

fivetran-avinash

@fivetran-reneeli A few comments before approval!

Co-authored-by: Avinash Kunnath <[email protected]>

CHANGELOG.md

fivetran-avinash

@fivetran-reneeli one final minor update but lgtm after that

Co-authored-by: Avinash Kunnath <[email protected]>

fivetran-reneeli added 7 commits January 30, 2025 23:10

update source type report

2f1fdc1

update app version report

046da4c

updates

632def4

seed file deletions and changes and additions

7084f63

update version and configs

c3a9b78

model revamps

5559078

subscription report

1ddd8c0

fivetran-reneeli commented Jan 31, 2025

View reviewed changes

models/apple_store__app_version_report.sql Outdated Show resolved Hide resolved

fivetran-reneeli added 12 commits January 31, 2025 16:23

rm int models

9b0b948

end model revisions

6746c2b

new int models

dd7458d

rename references

2bf9dfe

update yml with new fields, rm old fields, update uniqueness

c7df6f7

readme update and version bump

f941c5f

new int models

f61eafe

rm old int models

5c26fa4

updated reports, changelog, docs

77fd1f0

docs

5555efc

deps

105389c

schema

aeb2564

fivetran-reneeli requested a review from fivetran-joemarkiewicz February 1, 2025 06:33

fivetran-reneeli self-assigned this Feb 1, 2025

rm integrity territory report since staging model is removed, fix con…

e3e45db

…sistency test, update changelog

fivetran-joemarkiewicz requested changes Feb 3, 2025

View reviewed changes

models/apple_store__app_version_report.sql Outdated Show resolved Hide resolved

models/apple_store__app_version_report.sql Outdated Show resolved Hide resolved

models/apple_store__app_version_report.sql Outdated Show resolved Hide resolved

fivetran-reneeli added 5 commits February 4, 2025 10:26

seed updates

9fc03a1

style updates and regen docs

90753f0

rm active devices l30 days and update decision log. add date spine an…

43d4f81

…d update models

docs

cfafe69

switch from cross join to left join

4dc1890

fivetran-reneeli commented Feb 5, 2025

View reviewed changes

fivetran-reneeli and others added 5 commits February 11, 2025 17:30

Update dbt_project.yml

1e7c869

Co-authored-by: Joe Markiewicz <[email protected]>

Update README.md

afc2caa

Co-authored-by: Joe Markiewicz <[email protected]>

add upstream changes to doc, update territory def and logic, gen docs

4589fb7

rmm unnecessary coalesce

32e773e

changelog and deps

2a0ba4a

fivetran-joemarkiewicz reviewed Feb 11, 2025

View reviewed changes

packages.yml Outdated Show resolved Hide resolved

fivetran-reneeli and others added 6 commits February 11, 2025 18:47

Update packages.yml

1e57b4b

Co-authored-by: Joe Markiewicz <[email protected]>

temp change back deps

f27cd33

schema

42cfa02

modified seeds

cb9818f

change name of int model bs same as downsteam in app reporting

9dda153

updates, add bit about detailed vs standard, update min date

8076c46

fivetran-reneeli requested a review from fivetran-joemarkiewicz February 14, 2025 23:06

fivetran-joemarkiewicz requested changes Feb 19, 2025

View reviewed changes

update versions, changelog

deb51f8

fivetran-joemarkiewicz approved these changes Feb 19, 2025

View reviewed changes

fivetran-avinash reviewed Feb 19, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

fivetran-avinash reviewed Feb 19, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

fivetran-avinash reviewed Feb 19, 2025

View reviewed changes

fivetran-reneeli and others added 3 commits February 20, 2025 12:38

Update CHANGELOG.md

f2854cc

Co-authored-by: Avinash Kunnath <[email protected]>

Update CHANGELOG.md

36b64c9

Co-authored-by: Avinash Kunnath <[email protected]>

changelog

30d4c8a

fivetran-avinash reviewed Feb 20, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

fivetran-avinash approved these changes Feb 20, 2025

View reviewed changes

fivetran-reneeli and others added 2 commits February 20, 2025 13:49

Update CHANGELOG.md

98d739c

Co-authored-by: Avinash Kunnath <[email protected]>

update deps

43d2de2

fivetran-reneeli merged commit 4bb38b2 into main Feb 20, 2025
9 checks passed

fivetran-reneeli deleted the nov_2024_schema branch February 20, 2025 20:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nov 2024 schema #32

Nov 2024 schema #32

fivetran-reneeli commented Jan 31, 2025 •

edited

Loading

fivetran-joemarkiewicz left a comment

fivetran-reneeli Feb 5, 2025

fivetran-joemarkiewicz Feb 10, 2025

fivetran-joemarkiewicz Feb 11, 2025

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz Feb 19, 2025

fivetran-joemarkiewicz Feb 19, 2025

fivetran-joemarkiewicz Feb 19, 2025

fivetran-joemarkiewicz Feb 19, 2025

fivetran-reneeli Feb 19, 2025

fivetran-joemarkiewicz Feb 19, 2025

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz Feb 19, 2025

fivetran-avinash Feb 19, 2025 •

edited

Loading

fivetran-reneeli Feb 20, 2025

fivetran-avinash Feb 19, 2025 •

edited

Loading

fivetran-reneeli Feb 20, 2025

fivetran-avinash left a comment

fivetran-avinash left a comment


		{% set first_date_query %}

		select min(date_day) as min_date_day

		@@ -1,4 +1,14 @@
		# dbt_apple_store version.version
		# dbt_apple_store v0.5.0-a1

Nov 2024 schema #32

Nov 2024 schema #32

Conversation

fivetran-reneeli commented Jan 31, 2025 • edited Loading

PR Overview

PR Checklist

Basic Validation

Detailed Validation

If you had to summarize this PR in an emoji, which would it be?

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-avinash Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-avinash Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-avinash left a comment

Choose a reason for hiding this comment

fivetran-avinash left a comment

Choose a reason for hiding this comment

fivetran-reneeli commented Jan 31, 2025 •

edited

Loading

fivetran-avinash Feb 19, 2025 •

edited

Loading

fivetran-avinash Feb 19, 2025 •

edited

Loading